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The complexity of writing as a skill domain and the 
lack of consensus about its components have engendered much 
controversy about the type, length, or number of tasks that should be 
administered in a given test form and even about whether some aspects 
of composition require direct assessment through writing samples. 
Acknowledging this, a study was conducted to examine the 
comparability of writing competency profiles derived from test tasks 
differing in discourse and response mode. Two hundred high school 
students were given a multiple choice test and a paragraph writing 
task, as well as two full-length essay assignments. Ratings of the 
essays and paragraph on an analytic scale and scores on the objective 
test provided the bases for comparisons. Results indicated that 
levels of performance varied on tasks presenting different writing 
purposes. Also shown was that repeated applications of the scoring 
rubric produced measures that taped the same underlying content. In 
addition, factors reflecting the content of the writing subscale were 
strongly intercorrelated, a factor which is present no matter what 
response mode the subjects were assessed in. (HOD) 
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EFFECTS OF DISCOURSE AND RESPONSE MODE ON THE 
MEASUREMENT OF WRITING COMPETENCE 

As school district and state assessment programs attempt to test 
student basic skills achievement, attention to the methodological 
problems inherent in measuring writing competence increases. The com- 
plexity of writing as a skill domain and the lack of consensus about 
Its components have engendered much controversy abo-jt the type, length 
or number of tasks that should be administered for a given test form 
and even about whether some aspects of composition require "direct" 
assessment through th> elicitatlon of writing samples. 

Two salient measurement issues involved in specifying writing 
task types are the response mode (selected vs. constructed) and the 
discourse mode required by the tasks. Conventionally, large scale 
assessments have dealt with the response mode issue by measuring writ- 
ing skills indirectly with multiple choice tests. Support for such 
Indirect measurement derived from reported iiigh correlations between 
objective tests and direct measures of written production, from the 
erratic reliabilities accompanying impressionistical ly scored esssays 
and from the economic and logistical demands of collecting and scoring 
writing samples (Braddock, 1963; Godshalk, Swineford A Coffman, 1966; 
Breland, 1977). More recently, however, demands for writing tests 
with content, construct and ecological validity have prodded rein- 
statement of direct, written production tasks, Yet even when assess- 
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ments collect writing samples, students usually produce only one 
composition, despite the well documented fluctuation of writing 
performance from one sample to the next (French, 1962; Braddock, 1963; 
Diederich, 1974). By necessity, a single sample taps student 
performance in only one type of discourse and on one topic. This lim- 
itation presents a measurement problem since the generic methods of 
text development in particular forms of discourse differ substantially 
from one another. Salient structural features of argument, for in- 
stance, are issues, reasons, and conclusions, while stories include p- 
lot, character, setting, and theme. Exposition involves main idea, 
supporting detail and logical development; narrative elaborates events 
in chronological order, and description portrays concrete details in 
spatial order. Since different purposes set for writing tend to eli- 
cit the generic structural elements of the modes of discourse, 
(Kinneavy, 1971), it seems likely that the schema or frames activated 
in the writer by writing tasks varying in purpose should differ 
(Anderson, 1977; Minsky, 1976). 
Research on Discourse Mode Effects 

Evidence from reading and writing research support the distinc- 
tiveness of processes required by varying discourse models. Reading 
research suggests that different schema are used as students attempt 
to comprehend narrative and expository text (Meyer, 1975; Graesser, 
Hauft-Smith, Cohen & Pyles, 1979). As 1n reading, writers also employ 
various skills and personal resources to meet the demands intrinsic to 
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a kind of writing or mode of discourse. For students learning to 
write, these different discourse modes represent very dissimilar chal- 
lenges and, futhermore, attempts to compare student writing skills 
across modes of discourse constitute a real assessment issue. In one 
of the most frequently cited writing assesment studies, the perfor- 
mance variability in "topic" discussed by Godshalk, et al . cannot 
separate fluctuations due to the five different discourse modes stimu- 
lated by the assignments from that due to the subject matters ad- 
dressed. Veal and Tillman (1971) reported variability in elementary 
students 1 performance on tasks specifying different discourse aims, as 
did Praeter and Padia (1980). 

Moreover, other writing research demonstrates that different 
writing purposes lead writers to vary structural complexity (Crowhurst 
& Piche, 1979; Crowhurst, 1980; Perron, 1977) and to represent writing 
topics quite differently (San Jose, 1973; Perron, 1977). Most impor- 
tantly for instruction and evaluation, this accumulating body of 
writing research suggests that different writing purposes require 
dissimilar writing strategies. of unequal difficulty for individual 
students. Cooper (1979) has cited research indicating that sentence 
structures shift when discourse mode changes and speculated that a 
student's planning demands for an essay might change as much as 5C%. 

The implications for writing assessment of reading and writing 
studies on discourse mode effects is that the mode of discourse of the 
writing purpose will make a difference in writing performance. For 
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example, students might be more skilled at narrative writing tasks 
requiring chronological development than at expository tasks requiring 
logical development. Thus the profile of writing competence for a 
student based on a writing test calling for exposition may differ from 
the profile of writing competence for that same student on a narrative 
or persuasive task. 
Research on Response Mode Effects 

In addition to the question of skill commonality across discourse 
modes or genres, the question of the response mode or measurement form 
1n which the writing skill should be assessed continues as a hotly 
debated topic. While many claims are made for the predictive or 
concurrent validity of indirect, objective writing measures (Cqffman, 
1971; Breland, 1977), Indirect measures simply are not considered by 
writing researchers to meet the more crucial standards of content or 
construct validity (Braddock, et al . , 1963; Cooper & Odell, 1977). 
While selected responses elicited by multiple choice tests may provide 
valuable Information, they are, nonetheless, measures of processes 
required 1n reading comprehension, not measures of actual production 
ability. As such, responses to recognition tasks are often considered 
by learning theorists as, at best, behaviors enroute to constructed 
responses (Bourne, 1966, Skinner, 1963). 

Comparisons of direct and Indirect writing measures have yielded 
moderate correlations between scores" from the two response modes. In 
one of the seminal writing assessment studies, Godshalk, et al . (1966) 
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reported correlations from .46 to .75 between the sum of five essay 
scores from high school students and their College Board English 
Comprehension Test. In an attempt to validate ETS's Test of Standard 
Written English (TSWE) Breland, Conlan and Rogosa (1975) found 
correlations of only .42 between that mechanics-oriented test and a 20 
minute essay scored on a 4-point scale. In a subsequent study, 
Breland and Gaynor (1979) reported correlations ranging from .58 to 
.63 between students 1 three separate essay scores awarded on a 6-point 
scale and the TSWE, while the correlation between the sum of the three 
essays and the TSWE was .76. Similar low to moderate correlations 
of. 43 to .67 were found in a comparison of the American College 
Testing Program's English Usage Test (also emphasizing sentence-level 
skills) and students 1 scores on three essays. Kogan and Mishler's 
(1980) study of the relationship between third and eighth grade stu- 
dents 1 Metropolitan Achievement Test scores and one essay yielded cor- 
relations of .68 and. 65, while the correlations increased to .75 and 
.81 when a second essay score entered into the calculations. 

In general, these studies related essay scores based on norm- 
referenced holistic ratings to a total multiple choice test score, 
apparently assuming both sets of measures tapped the same set of 
writing skills. However, content analyses of the essay rating cri- 
teria reveal that, while often vaguely worded, they did reference 
whole-text features such as thesis, coherence, support and style, as 
well as sentence-level mechanical conventions. Items on the multiple 
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choice tests, on the other hand, often emphasized sentence-level mech- 
anics and required few if any text-level discriminations and, 
obviously, no production responses. At issue then is not simply 
whether measures correlate statistically, but whether different meas- 
ures focus on the same text featues of written productions and whether 
they reflect the same underlying skill constructs. 

In a study attempting to compare direct and indirect measures of 
reasonably parallel text features, Spooner-Smith (1978) used domain- 
referenced skill specifications to design multiple choice items analo- 
gous to essay rating criteria. She found correlations of the multiple 
choice test total score with a General Impression score of .65 and 
with the total of analytic ratings of .61. Relationships between 
analogous features such as Organization and Support, however, were 
much lower, ranging from .23 to .55. Her findings suggested that when 
multiple choice scores and essay scores derived from precisely matched 
definitions of text features, the. comparabil i ty of scores on these 
component writing skills might be even lower than those previously 
reported. 

Design Requirements for a Study of Discourse and Response Mode Effects 

To compare the information yield and psychometric quality of 
writing measures involving different discourse and response modes, 
data are needed that contrast the performance of a group of examinees 
across equivalently specified skill domains varying only by the modes 
of measurement. The specific test objectives (skill domains), stimu- 
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lus dimensions , instructions to examinees, and response criteria/ 
characteristics need to be matched as closely as possible across dis- 
course and response modes; i.e., each of the measures should present 
parallel content-valid procedures for assessing the same skill or 
skills. Data from measures designed to be psychologically parallel 
can then be examined for evidence of the construct validity and relia- 
bility of the discourse and response mode distinctions. A test of 
writing researchers 1 contentions that text-level writing skills such 
as thesis statement, organization and support are best measured by 
written production tasks would involve comparing multiple choice pas- 
sage comprehension measures of such subskills as organization, 
support, and mechanics with ratings of these features in text the 
student produces. This paper reports such comparisons for score 
profiles obtained from analytically scored direct assessments of 
student writing (essay and paragraph length writing samples) and an 
indirect assessment (multiple choice questions concerning prose 
passages )♦ Measures were designed to be conceptually parallel by 
using the domain specifications that guided development of 
directions/prompts for the writing tasks to construct the passages 
employed in the multiple choice task. Similarly, the dimensions of 
writing quality making up the analytical scoring rubric applied to the 
writing samples determined the specific aspects of the prose passages 
the multiple choice measure questioned. 
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The measurement issues addressed in the study concerned the com- 
parability of writing scores obtained from tasks varying in discourse 
and response mode. The study departed from more conventional metho- 
dology in two respects. First, the measures of writing skills in the 
different response modes were specifically designed to present tasks 
parallel on all dimensions but the discourse and response mode vari- 
ables. Second, the study augmented the standard correlational compar- 
isons of the measures with multitrait multimethod (MTMM) factor analy- 
ses testing specific hypotheses about sources of variation underlying 
student's writing score profiles derived from the alternative meas- 
ures. The analyses examined the convergent and discriminant validity 
of scores derived from different discourse and response modes, treat- 
ing the scale scores comprising the writing profiles as "traits," and 
the discourse or response modes as "methods" (Campbell & Fiske, 1957). 
Correlations among different operational izations of the same variable 
should arise from the influence of a single common factor or trait 
(e.g., organization). Also, the method of measurement should exert an 
Influence on each variable, so that different variables will covary to 
a greater degree when measured by a common method than when measured 
by different methods; this covariation can be thought of as reflecting 
the operation of a common method factor. This confirmatory factor 
analytic approach to MTMM validation has been implemented in other em- 
pirical studies (Joreskog, 1974; Traub & Fisher, 1977; Werts, Joreskog 
& Linn, 1972). Traub and Fisher (1977) for example, compared verbal 
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and quantitative scores derived from fill-in, right/wrong multiple- 
choice and partial knowledge multiple choice response formats in 
exactly this fashion. 

In this study we asked 1) whether student writing performance 
profiles are comparable on tasks differing in discourse mode (writing 
purpose), and 2) whether tasks requiring different response modes 
(paragraphs, essays, and multiple choice items) provide the same type 
and quality of information about student writing competence. In the 
MTMM framework, we examined whether distinctive common factors under- 
lay the corresponding variables from the writing profiles derived from 
the discourse and response modes variations. 

Method 

To examine the relationship of writing scores yielded by tasks 
differing on the two variables, discourse and response mode require- 
ments, high school students received writing tests on three separate 
occasions. Each student received a multiple choice test and a para- 
graph writing task, as well « two full length essay assignments. 
Ratings of the essays and paragraph on an analytic scale and scores on 
the objective test provided the bases for the comparisons. 
Sample 

Approximately two hundred eleventh and twelfth grade students 
from three high schools in a small school district 1n the Los Angeles 
area participated 1n the study. Students were selected whc were 
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attending English or composition classes that were judged by teachers 
to contain average or above average pupils. Scores from the verbal 
portion of the Differential Aptitude Test were available for 92 stu- 
dents in the sample; the mean percentile score for this subsample was 
63.9 (s.d. = 28.6). 
Design 

Students within each class were randomly assigned to one of four 
testing conditons defined by different discourse mode combinations for 
the essay tasks. In Conditions 1 and 2 (Same Genre), the three con- 
structed response writing tasks (two essays and one paragraph) were in 
the same discourse mode. Condition 1 students wrote two expository 
essays and an expository paragraph; Condi ton 2 students wrote two nar- 
rative essays and a narrative paragraph. 

In Conditions 3 and 4, (Different Genre) students wrote one nar- 
rative and one expository essay. Condition 3 students wrote an expos- 
itory essay on Topic A and a narrative essay on Topic B, while 
Condition 4 students wrote an expository essay on Topic B and a 
narrative essay on Topic A. Half of the subjects 1n Conditions 3 and 
4 wrote an expository paragraph, while half wrote a narrative para- 
graph. 

Response mode, the second design factor > was a within-subject 
factor and consisted of the multiple choice test (selected response), 
the paragraph (short constructed response) and the essay (long con- 
structed response). During the three testing occasions, subjects 
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received the multiple choice test and paragraph on one occasion, and 
an essay on each of the other two occasions. The design counterbal- 
anced the order in which students received the tasks. 
Measures 

The essay and paragraph tasks were constructed in accordance with 
a set of domain specifications for expository and narrative writing. 
These specifications included the purpose of the writing assignment, 
guidelines for appropriate topics, the response criteria by which 
written products were to be judged, and guidelines for the content and 
format of the directions for the tasks. The response criteria were 
chosen to reflect the discourse features of an analytic scoring system 
developed at UCLA (Pitts, 1978; Spooner-Smi th , 1978; Winters, 1978; 
Quellmalz, 1979). The version of the scoring system used 1n this 
study generated five ratings for each written product: 

(1) General Impression — A global judgment of writing quality 
assigned by raters after a quick initial reading of the 
writing sample. 

(2) Focus—The extent to which the subject and main idea of the 
writing sample were clearly stated or implied. 

(3) Organization — The extent to which the main Idea was developed 
according to a discernible method of organization (e.g., 
clear chronological or logical development). 

(4) Support — The extent to which generalizations and assertions 
were supported by specific, relevant, subordlnant statements. 
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(5) Mechanics—The extent to which the writing sample was free 
from intrusive sentence-level mechanical errors (e.g., usage, 
sentence construction, spelling, capitalization and punctua- 
tion). 

Each essay and paragraph was assigned ratings on these five subscales 
by one of two pairs of trained reader, the median general izabili ty 
coefficient for the two rater pairs were .61 and .83 across topics/ 
occasions and subscales. The three writing samples representing 
direct measurement (two essays and one paragraph) generated 15 sub- 
scale scores, each on a one (low) to four (high) scale. The scores 
were calculated by averaging the scores assigned by both raters to 
each written product for each subscale. 

The stimulus attributes from the specifications for the writing 
tasks were used to develop the passages to be read in the multiple 
choice task. Ten passages were constructed, five expository and five 
narrative. For each passage, there were three questions, designed to 
be analogous to text features included in the rating scales — main idea 
(focus), organization, and support. Main Idea questions were refer- 
enced to a stated generalization near the beginning or end of the 
passage. Organization questions required the selection of a new 
sentence that would best fit at a point in the passage marked by an 
arrow. Support questions asked which new sentence would best support 
the main Idea of the passage. 

Results 
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Discourse Mode Effects 

The first set of analyses compared students 1 scores according to 
the discourse mode of the task. 

Table 1 presents means and standard deviations of essay ratings 
for each of the four test conditions. 



Insert Table 1 here 



On all five subscales on on total essay scores, narrative ratings 
were lower than expository ratings* This finding may be due to the 
differential currlcular emphasis given to narrative and expository 
writing 1n the high schools, to subjects 1 lack of knowledge, at a 
personal experience level, required to deal with the narrative topics, 
or to raters 1 tendency to score narratives more stringently. 

Table 2 displays the correlations between students 1 two essay 
scores on each of the analytic scale subscales* As expected, 
correlations between essay scores for students writing two essays in 
the same discourse mode (Same Genre) arc higher than those for 
students 1n the Different Genre conditions* 



Insert Table 2 here 



An examination of the correlations for each subscale across con 
ditlons suggests that General Impression and Organization seem to 
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differentiate most between the different discourse modes (columns 3 
and 4 1n the table). This finding might also be expected, since Gen- 
eral Impression requires a judgment about the global quality of the 
essay as an example of exposition or narration. Therefore the con- 
stellation of essay factors influencing this judgment should be the 
most comprehensive for each discourse mode and thus the most discri- 
minating. Structurally, exposition and narration differ dramatically 
in their characteristic use of logical or temporal organizations, 
respectively. On the Mechanics subscale, correlations across condi- 
tions are most comparable, reinforcing the notion that the constella- 
tion of syntactic, punctuation, spelling and usage skills may not vary 
between modes of discourse so much as text-level skills do. 

Differences between the correlations pooled within same and dif- 
ferent genre conditions reveal that the relationship between student's 
two essay scores on General Impression, Organization, and the Total is 
significantly stronger when students write in the same genre than when 
they write in different genre. 

Only between-group comparisons were conducted between discourse 
modes for the paragraph data since each student wrote only one para- 
graph. Table 3 presents the results of these comparisons. The same 
analytic scales for narration and exposition used for rating the 
essays were also used for rating the paragraphs. 



Insert Table 3 about here 
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Subscale scores ranged from 1-4, total scores from 1-20. Ratings of 
narrative paragraphs were generally lower than ratings of expository 
paragraphs. Ratings of expository paragraphs differed significantly 
from narrative paragraphs on the General Impression, Focus, and Organ- 
ization subscales and on the Total scores. Consonant with essay data, 
Mechanics and Support were not as influenced by the different dis- 
course tasks. 

Multiple choice test comparisons of interest were the scores each 
individual received on the narratve and expository sections of the 
exam. Table 4 presents the means and standard deviations. 



Insert Table 4 about here 



On this reading comprehension test of items measuring recognition 
of writing-related skills, students were able to answer Focus/Main 
Idea and Support questions similarly well for both expository and nar- 
rative passages. On Organization questions, however, students had 
more difficulty in general (73% overall average), particularly with 
narrative organization (66%). Correlations between individual mul- 
tiple choice subscales across genre range from .47 to .49, with the 
between-genre total score correlation equalling .65; these figures 
suggest little differential sensitivity to genre for the subscales, 
comparing most closely to the within -genre correlations reported in 
Table 2. 
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In the aggregate, the preceding analyses contrasting students 1 
performance on writing tasks differing in discourse mode suggest that: 

1) students' writing skills vary in the different discourse modes, and 

2) discourse mode score variability seems to be differentially dis- 
tributed across writing subskills. These results occurred in separate 
analyses of students' essays, paragraphs and multi -choice scores. 

To test the effect of discourse mode on subskills, the data were 
then subjected to multi-trait multi-method (MTMM) analyses using con- 
firmatory factor analysis techniques (Joreskog, 1974). Traits were 
defined as the writing subskills; methods were defined as exposition 
and narrative. The analyses required within-subject measures of trait 
and discourse mode, therefore data from 94 subjects in test conditions 
3 and 4 were used to examine the effects of discourse mode. In these 
conditions students wrote an expository and narrative essay and 
answered multiple choice questions about expository and narrative pas- 
sages. Since students wrote both expository and narrative paragraphs, 
paragraph scores were not included in the analyses. 

To examine the factor structure of discourse modes in essay per- 
formance, the five analytic subscales of General Impression, Focus, 
Organization, Support and Mechanics formed the trait dimensions. The 
five trait indicators for each genre were formed by averaging scores 
over raters and standardizing across topics. Thus ten scores were 
constructed, two each (expository and narrative) for each of the five 
subscales. 



ERLC 



17 



To examine the factor structure of discourse modes across 
response modes, a second set of analyses used just the three subscales 
(traits) common to the essay and multiple choice tests, Focus, 
Organization, and Support* For the multiple choice test, number cor- 
rect scores were formed within discourse mode for each of the three 
multiple choice subscales. This second set of analyses employed six 
scores, two each (expository and narrative) for Focus, Organization 
and Support. 

Maximum likelihood estimates of the parameters of the MTMM con- 
firmatory factor analysis models were obtained from the LISREL compu- 
ter program for the analysis of covariance structures (Joreskog, 1973, 
1977; Joreskog 5 Sorbom, 1978). The LISREL program allows the analyst 
to treat model parameters (e.g., factor loadings or factor Intercor- 
relations) in one of three ways: (a) as free parameters to be esti- 
mated by the program; (b) as fixed parameters specified in advance to 
equal some fixed number (usually zero); or (c) as constrained para- 
meters to be estimated by the program subject to the constraint that 
they equal other estimated parameters. In addition, the program com- 
putes standard errors for all free and constrained parameters, as well 
as an overall chi square test of the model's fit to the data. All 
model equations (in LISREL notation) are of the form: 



Y = 



(1) 



= A V YA 



Y 



+ 9 



(2) 
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Each observed score in Y depends on the latent variables 5 and f> com ~ 
mon factors and measurement errors, respectively. Equation (2) shows 
the hypothesized structure underlying the covariance matrix of the 
Y's; it consists of a matrix of factor loadings 7 (hereafter Psi), 
and a (usually) diagonal matrix of error variances. Interest in the 
analyses to follow focuses on the contents of Lambda and Psi. 

The first set of MTMM analysis examined the influence of subscale 
content (i.e., Focus, Support, etc.) and essay genre (Expository and 
Narrative) on the scores for students writing essays in different dis- 
course modes. Eight scores are entered into the analysis, the measure 
of Focus, Organization, Support, and Mechanics for each essay Cor- 
relations among the scores are shown in Table 5, and a path diagram of 
the Initial factor analysis model in Figure 1, Panel A. 



Insert Table 5 and Figure 1 here 



In Panel B of the figure is shown the path diagram for the final model 
for these eight writing scores. In moving from the initial to the 
final model a number of intermediate analyses were run, each refining 
the factor structure until a completely adequate fit to the data was 
obtained. Parameter estimates for the final model are shown in Table 
6. 



Insert Table 6 about here 
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From the path diagram in Panel B, it can be seen that only two of 
the original four trait factors emerge in the final model, while 
"method 11 factors corresponding to discourse modes remain in the model. 
The one assymetry between discourse modes is the behavior of the 
Organization subscales: the measure of organization from the exposi- 
tory essay apears to be more or less fully saturated with genre varia- 
tion, which, when removed, leaves an insignificant remainder of reli- 
able variation to be shared with the other measures of coherence. The 
table shows the matrix of estimated factor loadings. All of the esti- 
mated parameters in the model are statistically different from zero. 
Except for Mechanics, all of the subscales contain relatively larger 
components of genre-related variation than "trait" variation (for both 
Support measures and Expository Organization, genre completely domi- 
nate). Variation in student Mechanics, on the other hand appears 
comparatively robust to the influence of genre. The factor intercor- 
relation matrix (Psi) is not included in the Table since all correla- 
tions are zero. While factor intercorrelations were permitted in the 
initial model, the factors which remain in the final solution are 
orthogonal . 

In fitting the MTMM models to the writing score data, we attempt- 
ed, where possible, to constrain appropriate pairs of factor loadings 
to equal one another. That is, it was usually of interest to check 
whether, for example, the loadings of the Focus variable derived from 
an Expository essay equalled those of the Focus variable from a Narra- 



ERIC 



22 



20 



tlve essay. We see in the table that such equality constraints were 
incompatable with the subscale variables' relations to the genre fac- 
tors (the first two columns of the Lambda matrix), while they are com- 
patable for subscale relations to the trait factors (where present). 
Finally, the table reports a non-significant goodness-of-fit statistic 
(chi square on 17 df = 18.678; £ = .347), indicating this model cannot 
be rejected. 

The overall picture of genre effects on writing presented by this 
analysis is that, except for relatively lower level skills (mechan- 
ics), the discourse mode students are required to write in is a strong 
influence on their performance. The different subskills Included in 
the scoring rubric seem definitely to interact with discourse mode 
and, at the same time, to varying degrees are independent sources of 
variation in student writing performance. 

Additional information was collected during the study which bears 
on possible genre effects on writing. For three of the four subscales 
Included in the previous analysis, multiple choice passage comprehen- 
sion subtest scores were available for Expository and Narrative pas- 
sages. We take up these six variables in the next set of analyses. 
With only six observed scores, it is not possible to specify a full 
MTMM model as in the previous analysis. Furthermore, one of our 
hypotheses concerns the adequacy of multiple choice measurements for 
obtaining distinctive information about writing subskills. With these 
concerns in mind, we specified three alternative factor models for the 
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multiple choice scores; these are displayed in Figure 2 as Panels A, 
8, and C. 

Insert Figure 2 about here 



Panel A shows a one-factor model, corresponding to the hypothesis that 
all of the multiple choice scores measure the same thing, e.g., general 
comprehension. Panel B shows a two-factor model, testing the possi- 
bility that the genre in which each of the passages is written deter- 
mines how the resulting scores covary. Lastly, Panel C shows a factor 
model with three traits, one for each subscale. Although these models 
cannot be strictly compared 1n a statistical sense (i.e., they are not 
hierarchically related to one another), gross differences 1n goodness- 
of-fit can guide the selection of an appropriate final model. Corre- 
lations among the multiple choice variables are shown 1n Table 7. A 
summary of the best fitting models 1s presented 1n Table 8. 



Insert Tables 7 and 8 here 



The two-factor model was found to fit less well than either ths 
one or three-factor models (p_ ■ 0.000), suggesting that, contrary to 
the picture of genre effects observed for actual writing tasks, pas- 
sage genre was not a source of discriminant validity for multiple 
choice scores. The one- and three-factor final models have identical 
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degrees of freedom and goodness-of -f it statistics, indicating that 
these are statistically equivalent models. In the one-factor model, 
this degree of fit was obtained by allowing residual covariation among 
the pairs of measurement errors associated with each trait— what is 
common among the scores is captured by the single common factor and 
what is unique to each trait by the residual covariances. In the 
three-factor model, the common variance and residual covarlance are 
combined Into three correlated factors. In a sense, then, we might 
equally well interpret the multiple choice scores as measuring three 
correlated factors. We are led tc ;?efer the latter interpretation by 
the fact that the standard errors for the estimates of factor inter- 
correlations in the three-factor model are sufficiently small as to 
make it highly improbable that each of these correlations would actu- 
ally be equal to unity, the situation that would need to obtain if a 
one factor model were appropriate. We should note that an additional 
two-factor model was tried, combining the focus and organization mul- 
tiple choice scores 1n"o a single Coherence factor, that produced 
a considerably poorer fit than the one- and three-factor models shown 
in Table 8. 
Response Mode Effects 

The second measurement issue addressed by the study was whether 
tasks requiring different response modes (direct production modes: 
essays, paragraphs; indirect selection modes: multiple choice) pro- 
vide the same type and quality of data about student writing abili- 
ties. 
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An MTMM factor analysis was performed to examine this issue. The 
"method" variables for response mode were constructed according to 
procedures analogous to those used to construct the discourse mode 
variables. Each of the five subscale scores provided by the essay and 
paragraph ratings was averaged over raters, then standardized within 
topic and discourse mode. Standardizations were employed to reduce 
possible Interactions between response mode and genre and/or topic. 
Number correct scores were formed within genre for each of the three 
multiple choice subscales, standardized within genre, summed across 
genre, and then restandardized to produce scores scaled 1n a manner 
comparable to those derived from the writing samples. No measures of 
Mechanics were Included in the multiple choice task. Complete data 
were available for 148 of the student*. In sum, 15 scores were con- 
structed for analysis, three measures of Mechanics (two essay and one 
paragraph), and four each of focus, Organization and Support (two es- 
say, one paragraph and one multiple choice). Table 9 presents the 
correlations matrix for the three response modes and four subscales. 



Insert Table 9 here 



The MTMM analyses began by considering the data for the two es- 
says each student wrote, i.e., "Essay 1" and "Essay 2" methods only; 
the first analysis included the eight scores defined for these two 
conditions, two measures of focus (fe^ fe 2 ), Organization (oej, oe 2 ), 
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Support (sej, se2), and Mechanics (me^, me2). The model specified for 
these variables includes the three relatively distinct trai t/subscale 
content factors emerging from the discourse mode analyses and two 
"method" factors, one for each essay. In keeping with the results 
from the final factor analysis of the discourse mode data, the focus 
and organization subscales are combined into a single trait/content 
factor, Coherence. Figure 3 displays a path diagram for this model 
and Table 10 presents the LISREL estimates of the free and constrained 
parameters for the model . 



Insert Figure 3 and Table 10 here 



As in the model for the discourse mode MTMM analysis, trait fac- 
tor intercorrelations are unconstrained in this model, while the 
method factors are constrained to be uncorrected with each other and 
with the subscale/trait factors. This latter restriction reflects our 
hypothesis that the method factors act as independent additive compo- 
nents influencing observed scores. In addition, trait factor loadings 
for pairs of subscale measures have equality constraints placed on 
them. These constraints test whether subscale scores from different 
essays exhibit the same degree of relationship to the traitfactor they 
measure. 

The model as a whole cannot be rejected; the non-significant chi 
square (p>^ .202) suggests that the model provides an adequate account 
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for the observed correlations among essay variables. Loadings of the 
essay variables on their respective trait factors are all substantial, 
ranging from a low of .472 for Organization to a high of .768 for 
Mechanics. Method factor loadings for the Organization subscale are 
relatively high while those for the Support subscale are moderate. 
Both Focus and Mechanics method loadings are low. In terms of the 
model 's decomposition of observed scores into trait and method compo- 
nents, then, Focus and Mechanics show stronger dependence on trait 
than method factors and Support is about equally related to both 
sources. Organization, however, is more strongly influenced by method 
variance, reflecting, in part, the genre differences for this subscale 
found in the earlier analyses. Turning to the psi matrix, estimates 
of the relations among the trait factors are moderate (below .80), 
ranging from a low of .626 for the correlation between Mechanics and 
Support to a high of .799 between Coherence and Support. Mechanics 
appears to be the most independent of the three content factors. The 
next MTMM model adds data from the paragraph task and expands to 12 
the number of variables in the analysis. Each of the four new sub- 
scale scores is specified to load on the same trait factor as did the 
analogous essay subscale in the previous analysis (no constraints are 
placed on these additional loadings); and a third method factor, Para- 
graph, is included to account for covariation specific to this mode of 
responding. Figure 4 displays the path diagram for this model and 
Table 11 presents the results of the LISREL estimation of the param- 
eters of this model . 
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Insert Figure 4 here 



Insert Table 11 here 



The model provides an adequate overall fit to the observed inter- 
correlations among the essay and paragraph variables (chi square on 43 
df * 51.533, £ < .175). This 1s consistent with the hypothesis that 
the scores generated by application of the rating system to paragraph 
length writing samples can be Interpreted as measuring the same under- 
lying content as the scores derived from full length essays. Inspec- 
tion of the lambda matrix shows that the loadings for paragraph 
subscale scores on their associated trait factors are of substantial 
magnitude 1n each case, and that the loadings on the paragraph factor 
follow the same general pattern as for the two essay method factors. 
With one exception, the paragraph variables appear to relate to trait 
factors less strongly than do the essay scores. The one exception 1s 
an interesting one: "sp M provides a clearer definition of the Support 
factor than either of the? support measures derived from essays. This 
would seem to suggest that the rater's task of judging the use of 
support is carried out more distinctly 1n the context of single para- 
graphs than it 1s in longer writing samples. 
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As for the essay-only model, the trait intercorrelatlons for the 
paragraph and essay model are moderate to high, Indicating consider- 
able Interdependence among the subscales. Again, Mechanics exhibits 
lower levels of relationship to the other subscales. 

Comparison of these two response mode models reveals two main 
differences. First, there is some Instability in the size of the 
essay variables' loadings on the associated trait factors as we move 
from the first to the second model. This leads to the interpretation 
that the factors composed of both essay and paragraph variables do not 
measure precisely the same content as factors composed of essay vari- 
ables only. Second, estimates of the trait Intercorrelatlons in the 
second model are greater than their counterparts 1n the first model. 
Thus, although the Inclusion of paragraph scores may have broadened 
the content of the trait factors, it seems also to have diminished 
their distinctiveness. 

The third MTMM analysis builds on the previous two by adding the 
three scores derived from the multiple choice items administered to 
the students in the study. Recall that only items analogous to the 
Focus, Organization and Support subscales were included 1n the mul- 
tiple choie test. This model differs from the previous one, then, by 
the specification of trait loadings for these three subscales, and the 
addition of a multiple choice method factor. Figure 5 displays the 
path diagram and Table 12, the LISREL estimates of the model parame- 
ters, for the analysis of data from all three response modes. 
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Insert Figure 5 and Table 12 here 



As in the first two analyses, the model provides a reasjnably 
good fit to the data (chi square on 76 df « 84.952, j> » .226), Imply- 
ing that the same 3-trait structure is not violated by the inclu<^~ 
of the multiple choice scores. By and large, however, estimates of 
the essay trait factor loadings have dropped in value in comparison 
with the corresponding estimates from previous models. Also, the 
trait factor intercorrelations have increased for Coherence and Mech- 
anics, indicating that the subscale content factors have drifted 
closer together as a result of adding the multiple choice variables. 
Thus, while the multiple choice scores apparently share some content 
with the constructed response variables to which they are purportedly 
analogous, they also seem to possess a higher degree of "latent col- 
linearit;*" (Yates, 1979) in the trait factor space. Whether this 
situation arises because the multiple choice variables are related to 
writing ability in some non-specific fashion, or because all of the 
variables, but especially the multiple choice scores, share a common 
dependence on general verbal ability, cannot be disentangled without 
additional analyses, including tests marking general ability factors. 
In any event, it is reasonable to interpret the Increased interdepen- 
dence among trait factors as an indication that the multiple choice 
scores possess generally lower validity as indices of distinctive 
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components of writing ability than do measures based on actual writing 
samples. 

The final model examines the relationship of the paragraph and 
multiple choice variables to the set of trait factors defined solely 
on the basis of the essay variables. Generally speaking, both the 
contribution of the essay scores to the definition of the subscale 
content factors and the degree of Independence of these factors from 
one another were reduced as data from the alternative response modes 
were added to the analysis. In the final model the trait factor 
structure obtained when only essay scores were Included 1n the analy- 
sis (cf., Figure 3 and Table 10) as a criterion definition of the con- 
tent underlying the subscales, treating the earlier trait factors as 
"unmeasured" criterion variables against which to compare the scores 
from the other two response modes. This can be accomplltshed 1n 
LISREL by modifying the specification for the third model (see Table 
12) in two ways. First, Instead of estimating trait loadings for 
essay variables, new specifications fix their values to equal those 
estimated from the essay data alone. Second, we place a similar con- 
straint on the trait factor intercorrelatlons 1n Psi, by fixing their 
values at those obtained in the essay-only solution. These two sets 
of restrictions will ensure that the essay-only trait factors will be 
reproduced exactly, and the standing of the paragraph and multiple 
choice variables can be evaluated vis-a-vis the essay criterion trait 
structure. The LISREL estimates of the free parameters of the final 
model are contained in Table 13. 
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Insert Table 13 here 



The only parameter estimates of direct interest in Table 13 are 
the trait factor loadings for the paragraph and multiple choice vari- 
ables. The data indicate near uniform reduction in their magnitude 1n 
comparison to the estimates obtained when trait factors are fit to 
data from all three response modes. This shift does not reduce the 
overall model fit (chi square with 83 df * 95.547, £ = .164). In all 
but two instances, paragraph and multiple choice trait factor loadings 
are lower than the corresponding loadings for the essay variables. 
Both exceptions are recurrences of the findings from the second and 
third models that the measure of Support derived from a paragraph 
length writing sample outperforms the Support measures based on full 
length essays and the Organization score in multiple choice 1s slight- 
ly more distinct than the Organization measures on essays respec- 
tively. Support, as measured by multiple choice items, seems to 
reflect relatively little of what is measured 1n actual writing 
samples; while multiple choice measures of Focus and Organization seem 
to convey a roughly comparable amount of Information about subscale 
content to that contained 1n a single paragraph. 
Summary and Conclusions 

The purpose of the study was to examine the comparability of 
writing competency profiles derived from test tasks differing in 
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discourse and response mode* Theory and research in the fields of 
learning, Instruction and rhetoric have fueled contentions that the 
knowledge structures and processing strategies activated by different 
writing aims and modes of responding are quite distinct. We were 
attempting to demonstrate the robustness of these claims from a mea- 
surement perspective. 

In practice, many current writing assessment programs fail to 
consider the validity of test data that does not distinguish between 
the demands of types of writing tasks and between the requirements of 
production and selection. At heart, the Issue is one of construct 
validity, do these alternative task and processing variables measure 
the same thing? Our results indicate that the answer 1s H no. H 

In this study the results of correlational, parametric and multi- 
trait multl -method analyses indicate that levels of performance vary 
on tasks presenting different writing purposes. These data cast doubt 
on the assumption that "a good writer is a good writer" regardless of 
the assignment. The implication is that writing for different aims 
draws on different skill constructs which must therefore be measured 
separately to avoid erroneous, invalid interpretations of performance. 
The findings suggest that generalizations about student writing compe- 
tence must reference the particular discourse domain rather than the 
general domain of writing. 

The study also investigated the distinctiveness of information 
about writing compoetence provided by direct and indirect measurement 
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techniques. Here again, the Issue 1s one of validity—do both 
response modes measure the same skill construct? Our results suggest 
this question cannot be answered unequivocally. Tests of response 
mode effects within MTMM confirmatory factor-analytic framework show 
method variance to be present in varying degrees. Organization 1s 
particularly influenced by the response requirements of the task 
(organization was also most sensitive to genre). The patterning of 
subscales' saturation with method variance 1s consistent within con- 
structed response tasks. Writing variables measured 1n a multiple 
choice format, however, follow a different pattern of method variation 
and, while no more strongly saturated with systematic error, show gen- 
erally lower levels of communal 1ty (i.e., more random measurement 
error). 

While models can be fitted to the data from all three response 
modes that confirm the subscales 1 content, the degree of Independence 
of the resulting subscale factors appears to be affected by which 
response modes are Included 1n the analysis. The most differentiated 
subscale factor structure is obtained by Including only essay vari- 
ables in the analysis; interdependence among the subscale factors 
increases with the addition of both paragraph and multiple choice 
measures. Thus, the effect of shortening the assessment task for the 
examinee through examination of just paragraphs or of changing the 
form of the response (multiple choice tasks) does not simply Increase 
the measurement error. The savings in testing time are obtained also 
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at the cost of clarity and distinctiveness 1n the Information about 
each of the subscales. When the subscale content factors are located 
1n the variable space so as to maximize their relationship to scores 
derived from the essay response mode, most other subscale-response 
mode combinations provide weaker substantive information. The main 
exception 1s the measure of Support based on paragraph-length writing 
samples which seems to be superior to the corresponding essay vari- 
ables 1n Its ability to capture subscale content. It may be that the 
use of support 1s less equivocally evaluated in the context of a sin- 
gle paragraph than 1n an essay containing multiple paragraphs, each of 
which may suggest a different view of the examinee's ability to pro- 
vide supporting detail. 

The MTMM analyses also provided Information about the validity of 
the rating scales. The MTMM analyses suggested, first, that repeated 
applications of the scoring rubric developed in our reseach to writing 
samples 1n fact produce measures that tap the same underlying content. 
Thus, given multiple measures of each subscale, 1t 1s possible to fit 
a factor analysis model that confirms their hypothesized content. 
Second, it was found that factors reflecting the content of the writ- 
ing subscales are strongly Intercorrelated, and this interdependence 
appears to be present no matter what response mode subjects are 
assessed 1n. When the global judgment for General Impression was 
removed and Focus and Organization combined into a Coherence subscale, 
scale 1ntercorrelat1ons became more moderate and distinct. Since 
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techniques for producing writing that is coherent, supported and mech- 
anically correct are often taught separately, further examination of 
the value of rating writing accoring to separate component features 
should consider both their diagnostic utility and component distinc- 
tiveness. 

Finally, the study exemplified the contribution HTMM analyses can 
make to validity studies. The technique may provide more sensitive, 
precise statistical indices of hypothesized competencies underlying 
test performance. 

In summary, the study highlights the importance of precision in 
designing, analyzing and reporting writing assessment data. It may be 
that the techniques developed for specifying domain -referenced skill 
boundaries can provide a reasonable framework or focusing attention, 
discussion, assessment and instruction on clearly bounded classes of 
writing performance. 
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Footnote 

^ote that we have excluded from these and subsequent analyses 
the General Impression ratings. In preliminary factor analyses 
including these scores , we found them to be inseparable from Focus and 
Organization ratings, suggesting that raters' general impressions con- 
tain little or no additional information. 
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Table 1 



Means and Standard Deviations of Essay Scores 



j est Same Genre Different Genre 

Condition 12 3 4 



Topic 




A 


B 




A 


B 




A 




B 


A 


B 


General 


X 


2 


20 


2 


.01 


1 


.27 




.88 


i 


.43 


1 


.89 


■..94 


2.09 


Impression 


sd 




60 




.70 


1 


.44 


1 


.09 


i 


.89 




.66 


.88 


.54 


Focus 


X 


2. 


45 


2 


.30 


2 


.21 


2 


.21 


2 


.28 


2 


.19 


2.26 


2.33 


sd 




69 




.64 




.68 




.64 




.59 




.69 


.51 


.53 


Organization 


X 


2. 


16 


1 


.98 


1 


.88 


1 


.62 


1 


.88 


1 


.95 


1.52 


2.08 


sd 




72 




.70 




.98 




.71 




.85 




.70 


.56 


.56 


Support 


X 


2. 


34 


2 


.38 


2 


.42 


2 


.19 


2 


.36 


2 


.28 


2.10 


2.26 


sd 




64 




.54 




.81 




66 




.70 




.65 


.54 


.53 


Mechanics 


X 


2. 


35 


2 


34 


2 


.56 


2 


42 


2 


28 


2 


.03 


2.35 


2.42 


sd 




64 




68 




.69 




67 




.80 




.81 


.56 


.54 


Total 


X 


11. 


50 


10 


90 


10 


35 


9 


32 


10 


23 


10 


.31 


9.18 


11.19 


sd 


2. 


78 


2 


67 


3 


84 


2 


89 


3 


36 


3 


.02 


2.02 


2.05 


h = 






40 




40 




39 




39 




40 




40 


54 


54 
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Table 2 





Correlation Between Students' Two Essays 






Same Genre 


Different 


Genre z 


difference! 


Condition 


1 


2 


3 


4 


142 vs. 3&4 


General 
Impression 


.56 


.39 


.33 


-.08 


2.52* 


Focus 


.43 


.68 


.41 


.37 


1.50 


Organization 


.42 


.42 


.20 


-.10 


2.55* 


Support 


.27 


.28 


.07 


.38 


.31 


Mechanics 


.58 


.68 


.63 


.50 


.65 


Total 


.60 


.55 


.41 


.22 


2.10* 


n = 


40 


39 


40 


54 




^Difference between 
3 and 4 after r-to-z 


correlations for 
transformation. 


conditions 


1 and 2 vs. 


conditions 



*p_>.05 



0 
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Table 3 

Difference Between Scores on Expository and Narrative Paragraphs 







Expository 


Narrative 


t diff. 


General 


X 


1.93 


1.05 


6.74*** 


Impression 












s.d. 


.71 


1.06 




Focus 


X 


2.34 


2.09 


2.93** 




s.d. 


.54 


.63 




Organization 


X 


1.95 


1.70 


2.41** 




s.d. 


.62 


.78 




Support 


X 


1.94 


2.02 


.87 




s.d. 


.67 


.71 




Mechanics 


X 


2.35 


2.29 


.65 




s.d. 


.65 


.74 




Total 


X 


10.50 


9.15 


3.37** 






n = 111 


n = 89 





**£_< .01 
***£ < .001 
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Table 4 

Means and Standard Deviations of Multiple-Choice Test Scores 







Expository 


Narrative 


Total 


Focus 


1 


4.61 (92%) 


4.50 (90%) 


9.13 (91%y 






77 


89 


1.38 


Organization 


T 


4.05 (81%) 


3.31 (66%) 


7.39 (73%) 




s.d. 


1.03 


1.13 


2.03 


Support 


7 


4.56 (91%) 


4.41 (88%) 


8.97 (90%) 




s.d. 


.77 


.98 


1.51 


Total 


X 


13.23 (88%) 


12.23 (82%) 


25.33 (84%) 




s.d. 


2.02 


2.43 


4.26 



n = 241 
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Table 5 

Correlations Among Essay Writing Scores 
From Different Discourse Modes 



FON 


FON 
1.00 


ORN 


SUN 


MEN 


FOE 


ORE 


SUE MEE 


ORN 


.690 


1.00 












SUN 


.474 


.600 


1.00 










MEN 


.351 


.286 


.322 


1.00 








FOE 


.327 


.330 


.060 


.222 


1.00 






ORE 


.108 


.007 


.116 


.125 


.456 


1.00 




SUE 


.066 


-.016 


.087 


.297 


.461 


.478 


1.00 


MEE 


.083 


.015 


.109 


.545 


.386 


.354 


.410 1.00 



N « Narrative 
E « Expository 
F * Focus 
0 * Organization 
S « Support 
M * Mechanics 
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Table 6 



LISREL Estimates for Final MTMM Model for Essay Writing 
Scores from Different Discourse Modes 



Lambda Matrix* 





Coherence 


Mechanics 


Narrative 


Expository 




.528 2 


0.0 


.562 


0.0 


FOE 


.5282 


0.0 


0.0 


.717 


ORN 


.607 


0.0 


.696 


0.0 


ORE 


0.0* 


0.0 


0.0 


.682 


SUN 


0.0 


0.0 


.866 


0.0 


SUE 


0.0 


0.0 


0.0 


.687 


MEN 


0.0 


.657 3 


.346 


0.0 


MEE 


0.0 


.657 3 


0.0 


.437 



iAll estimated coefficients exceed twice their standard errors. 
2 > 3 Pa1rs of loadings constrained equal. 
♦All zero coefficients fixed a priori . 
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Table 8 

LISREL Estimates for One- and Three-factor Models for Multiple 
Choice Variables from Different Discourse Modes 



l-Factor 


Lambda 


Residual Covariance 


FMN 


.638 


FMN x FME 


.068 


FME 


.638 






OMN 


.578 


OMN x OME 


.155 


OME 


.578 






SMN 


.631 


SMN x SME 


.095 


SME 


.631 


X ? - = 21. 

9 


167 (p < . 


3-Factor 


Lambda 








Focus 


Organi zatlon 


Support 


FMN 


.689 


0.0 


0.0 


FME 


.689 


0.0 


0.0 


OMN 


0.0 


.699 


0.0 


OME 


0.0 


.699 


0.0 


SMN 


0.0 


0.0 


.703 


SME 


0.0 


0.0 


.703 




PS1 








Focus 


Organization 


Support 


Focus 


1.0 






Organization 


.766 


1.0 




Support 


.832 


.743 


1.0 



X 2 = 21.167 (p < .012) 

9 ~ 



9 
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Table 9 

Correlation Matrix of Scores Containing Response Mode and Subscale Trait Effects 



FOE! 

ORE! 

SUEi 

MEEi 

FOE 2 

ORE 2 

SUE 2 

MEE 2 

FOP 

ORP 

SUP 

MEP 

FOMC 

ORMC 

SUMC 



FOEi 


OREi 


SUEi 


* ILL 1 


FOFo 
r ut 2 


nRFo 

urcc 2 


sue* 
out 2 


net 2 


CfiD HDD 

rUr UKr 


ci in 
5UP 


urn 
NEP 


P0MC 


1.000 
























.441 


1.000 






















.438 


.566 


1.000 




















.423 


.437 


.396 


1.000 


















.424 


.211 


.247 


.234 


l ooo 
















.345 


.252 


.230 


.264 


.608 


1 000 














.233 


.247 


.314 


.266 


.475 


. 550 


1 000 












.365 


.276 


.299 


.589 


.512 


.419 


.361 


1 000 










.231 


.057 


.252 


.305 


.270 


.192 


.186 


.353 


1.000 








1 Ofi 
• 1 jO 


• I/O 


000 

. its 


ooo 


.279 


.228 


.170 


.297 


.559 1.000 






.324 


.269 


.303 


.297 


.375 


.322 


.294 


.428 


.520 .579 


1.000 




.345 


.311 


.314 


.603 


.352 


.340 


.272 


.555 


.433 .429 


.330 


1.000 


.256 


.226 


.149 


.222 


.379 


.353 


.215 


.373 


.362 .286 


.344 


.311 


1.000 


.229 


.142 


.181 


.285 


.360 


.307 


.256 


.344 


.304 .339 


.328 


.352 


.474 


.239 


.312 


.197 


.317 


.264 


.253 


.172 


.354 


.259 .291 


.245 


.360 


.419 
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Table 10 

LISREL Estimates for Response Mode and Trait Effects 

1n Essay Data 

Lambda Matrix* 





Coherence 


Support 


Mechanics 


Essayx 


Essay 2 


Fe! 


.633 


0 


0 


.255 


0 


Fe 2 


.633 


0 


0 


0 


.485 


Oej 


.472 


0 


0 


.671 


0 


0e2 


.472 


0 


0 


0 


.639 


S ei 


0 


.535 


0 


.534 


0 


Se 2 


0 


.535 


0 


0 


.507 


Me! 


0 


0 


.768 


.289 


0 


Me 2 


0 


0 


.768 


0 


.254 



PSI C S M 



C 1.0 

S .799 1.0 

M .706 .626 1.0 

X* - 16.940 (p < .202) 
13 — 

*A11 estimated coefficients exceed twice their standard errors. 



Table 11 

LISREL Estimates for Response Mode and Trait Effects 
1n Essay and Paragraph Data* 

Lambda Matrix 





C 


s 


H 


El 


^2 


Paragraph 


F ei 


.581 


0 


0 


• 282 


0 


0 


Fe 2 


/581 


0 


0 


0 


.495 


0 


Fp 


.485 


0 


0 


0 


0 


• 473 


Oej 


.468 


0 


0 


• 683 


0 


0 


0e 2 


.468 


0 


0 


0 


.650 


0 


Op 


.417 


0 


0 


0 


0 


.780 


ie l 






n 

U 


• <JUC 


o 


o 


Se 2 


0 


.522 


0 


0 


.461 


0 


SP 


0 


.639 


0 


0 


0 


.421 


Me 1 


0 


0 


.773 


.250 


0 


0 


Me 2 


0 


0 


.773 


0 


• .192 


0 


Hp 


0 


0 


.728 


0 


0 


.207 



PSI C S M 

c To 

S .937 1.0 

M .809 .684 1.0 

7? « 51.333 (p 1 .175) 

_Jl2 

♦All estimated coefficients exceed twice their standard errors. 
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Table 12 



LISREL Estimates for Response Mode and Trait Effects in Essay, 
Paragraph and Multiple Choice Data 

Lambda Matrix 





c 


S 


M 


El 


^2 


P 


Multiple Choice 




.585 


0 


0 


.323 


0 


0 


0 


Fe 2 


.585 


0 


0 


0 


.472 


0 


0 


Fp 


.514 


0 


0 


0 


0 


.444 


0 


Frac 


.546 


0 


0 


0 


0 


0 


.399 


Oej 


.483 


0 


0 


.672 


0 


0 


0 


0e 2 


.483 


0 


0 


0 


.619 


0 


0 


Op 


.470 


0 


0 


0 


0 


.737 


0 


Omc 


.528 


0 


0 


0 


0 


0 


.462 


Sei 


0 


.469 


0 


.546 


0 


0 


0 


Se 2 


0 


.469 


0 


0 


.480 


0 


0 


SP 


0 


.'611 


0 


0 


0 


.398 


0 


Smc 


0 


.483 


0 


0 


0 


0 . 


.397 


Me! 


0 


0 


.772 


.277 


0 


0 


0 


Me 2 


0 


0 


.772 


0 


.183 


0 


0 


Mp 


0 


0 


.747 


0 


0 


.182 


0 


PSI 


c 


S 


M 










C 


1.0 














s 


.978 


1.0 












M 


.786 


.786 


.786 











*ij 6 - 84.962 (p <. .226) 



*A11 estimated coefficients exceed twice their standard errors. 
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Table 13 

LISREL Estimates for Response Mode and Trait Effects 
Using Essay Trait Factors as Criteria* 

Lambda Matrix 





C 


S 


M 


El 


E2 


P 


Mc 


Fei 


.6331 


0 


0 


.333 


0 


0 


0 


Fe 2 


.633 1 


0 


0 


0 


.436 


0 


0 


Fp 


.463 


0 


0 


0 


0 


.457 


0 


Fmc 


.526 


0 


0 


0 


0 


0 


.407 


Oej 


.472 1 


0 


0 


.676 


0 


0 


0 


0e 2 


.472 1 


0 


0 


0 


.606 


0 


0 


Op 


.408 


0 


0 


0 


0 


.767 


0 


Omc 


.492 


0 


0 


0 


0 


0 


.452 


Se l 


0 


.535 1 


0 


.538 


0 


0 


0 


Se 2 


0 


.535 1 


0 


0 


.496 


0 


0 


Sp 


0 


.604 


0 


0 


0 


.414 


0 


Smc 


0 


.410 


0 


0 


0 


0 


.475 


m x 


0 


0 


.768 1 


.290 


0 


0 


0 


Me 2 


0 


0 


.768 1 


0 


.178 


0 


0 


Mp 


0 


0 


.719 


0 


0 


.197 


0 


83 " 


95-547 (p < 0.164) 











♦All estimated coefficients exceed twice their standard errors. 
lCoeffldents fixed at values from Table 10. 
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FIGURE 1: Path Diagrams for MTMM Factor Analyses 
of Genre Effects on Essay Data 



Panel A 




Panel B 
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Figure 2: Path Diagrams for M7MM Factor Analyses 
of Multiple Choice Writing Data 



Panel A 




9 
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Figure 4 



Path Diagram for MTMM Model of Response .jde and Trait Effects 
in Essay and Paragraph Data 




ERIC 
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Figure 5 

Path Diagram for Models of Response Mode and Trait Effects 
in Essay, Paragraph and Multiple Choice Data 




